Experimentation at Twitter

Author: Robert Chang @_rchang

Why experimentation?

A/B testing in Obama’s Presidential Campaign

images/splash_page.png

Variation 1

images/obama_v1.png

Variation 2

images/obama_v2.png

Monumental Successes!

images/splash_result.png

Overview

Experimentation is at the heart of Twitter’s product development cycle:

Changing the Favorite Icon From Start to Heart

images/twitter_1.png

Improve the Relevance of Who to Follow Algorithm

images/twitter_3.png

Adding a Recap Module to the Home Timeline

images/twitter_4.png

How to run an experiment?

images/mental_model.png

Gather Samples

Bucket Size Estimator

images/bucket_es.png

Assign Buckets

Bucket Health

images/bucket_health.png

Apply Treatment

Measure Outcome

Adding Track Events

images/trackevent.png

Make Comparison

The probelm of early peaking

images/airbnb_example.png

Limitation of A/B Testing

Thank you!

/

#